Spark RDD常用算子操作（十） PairRDD的Action操作countByKey, collectAsMap

发表于 | 分类于 | 阅读次数：

原文作者：翟开顺
首发：CSDN
本人仅为自己方便查阅做了摘抄，请支持原作者
原文地址：https://blog.csdn.net/t1dmzks/article/details/72077428

PairRDD的Action操作countByKey, collectAsMap

countByKey

def countByKey(): Map[K, Long]
以RDD{(1, 2),(2,4),(2,5), (3, 4),(3,5), (3, 6)}为例 rdd.countByKey会返回{(1,1),(2,2),(3,3)}
scala例子

1
2
3
4
scala> val rdd = sc.parallelize(Array((1, 2),(2,4),(2,5), (3, 4),(3,5), (3, 6)))

scala> val countbyKeyRDD = rdd.countByKey()
countbyKeyRDD: scala.collection.Map[Int,Long] = Map(1 -> 1, 2 -> 2, 3 -> 3)

java例子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
    JavaRDD<Tuple2<Integer, Integer>> tupleRDD =
            sc.parallelize(Arrays.asList(new Tuple2<>(1, 2),
            new Tuple2<>(2, 4),
            new Tuple2<>(2, 5),
            new Tuple2<>(3, 4),
            new Tuple2<>(3, 5),
            new Tuple2<>(3, 6)));
    JavaPairRDD<Integer, Integer> mapRDD = JavaPairRDD.fromJavaRDD(tupleRDD);
    //countByKey
    Map<Integer, Object> countByKeyRDD = mapRDD.countByKey();
    for (Integer i:countByKeyRDD.keySet()) {
        System.out.println("("+i+", "+countByKeyRDD.get(i)+")");
    }
/*
输出  
(1, 1)
(3, 3)
(2, 2)

*/

collectAsMap

将pair类型(键值对类型)的RDD转换成map, 还是上面的例子

scala例子

1
2
3
4
scala> val rdd = sc.parallelize(Array((1, 2),(2,4),(2,5), (3, 4),(3,5), (3, 6)))

scala> rdd.collectAsMap()
res1: scala.collection.Map[Int,Int] = Map(2 -> 5, 1 -> 2, 3 -> 6)

java例子

1
2
3
4
5
6
7
8
9
10
JavaRDD<Tuple2<Integer, Integer>> tupleRDD =
        sc.parallelize(Arrays.asList(new Tuple2<>(1, 2),
        new Tuple2<>(2, 4),
        new Tuple2<>(2, 5),
        new Tuple2<>(3, 4),
        new Tuple2<>(3, 5),
        new Tuple2<>(3, 6)));
JavaPairRDD<Integer, Integer> mapRDD = JavaPairRDD.fromJavaRDD(tupleRDD);

Map<Integer, Integer> collectMap = mapRDD.collectAsMap();

0%